<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>GROOPS - Instrument data handling</title>

    <!-- JQuery and Popper -->
    <script src="https://code.jquery.com/jquery-3.4.1.slim.min.js" integrity="sha384-J6qa4849blE2+poT4WnyKhv5vZF5SrPo0iEjwBvKU7imGFAV0wwj1yYfoRSJoZ+n" crossorigin="anonymous"></script>
    <script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.0/dist/umd/popper.min.js" integrity="sha384-Q6E9RHvbIyZFJoft+2mJbHaEWldlvI9IOYy5n3zV9zzTtmI3UksdQRVvoxMfooAo" crossorigin="anonymous"></script>

    <!-- Bootstrap -->
    <!-- https://getbootstrap.com/docs/4.1/examples/ -->
    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/js/bootstrap.min.js" integrity="sha384-wfSDF2E50Y2D1uUdj0O3uMBJnjuUD4Ih7YwaYd1iqfktj0Uod8GCExl3Og8ifwB6" crossorigin="anonymous"></script>
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css" integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous">

    <!-- Mathjax -->
    <script type="text/x-mathjax-config">
    MathJax.Hub.Config({tex2jax: {inlineMath: [ ['$','$'] ],
                                  displayMath: [ ["\\[","\\]"] ],
                                  processEscapes: true},
                        TeX:     {Macros: {M: ["{\\mathbf #1}",1]},
                                  equationNumbers: {autoNumber: "all"}  }});
    </script>
    <script async src="https://cdn.jsdelivr.net/npm/mathjax@2.7.7/MathJax.js?config=TeX-AMS_CHTML" integrity="sha384-e/4/LvThKH1gwzXhdbY2AsjR3rm7LHWyhIG5C0jiRfn8AN2eTN5ILeztWw0H9jmN" crossorigin="anonymous"></script>

    <!-- lunr -->
    <script src="https://cdn.jsdelivr.net/npm/lunr@2.3.8/lunr.min.js" integrity="sha384-vRQ9bDyE0Wnu+lMfm57BlYLO0/XauFuKpVsZPs7KEDwYKktWi5+Kz3MP8++DFlRY" crossorigin="anonymous"></script>

    <!-- Mustache -->
    <script src="https://cdn.jsdelivr.net/npm/mustache@4.0.1/mustache.min.js" integrity="sha384-0PLEZVBpOQ+Kqw3anJWSNWvRxpEFt02tSpBvyRsA4WcvX/OTldWdXxGLVLvh954H" crossorigin="anonymous"></script>

    <!-- GROOPS Stylesheet -->
    <link rel="stylesheet" href="static/groops.css"/>

    <!-- icon -->
    <link rel="icon" href="static/groops_icon.png">

</head>
<body>
    <header>
        <nav class="navbar navbar-expand-lg navbar-dark bg-dark">
            <div class="container">
              <a class="navbar-brand" href="index.html"><img class="logo" src="static/groops_white.svg"></a>
              <div class="collapse navbar-collapse" id="mainNavbar">
                <ul class="navbar-nav mr-auto">
                  <li class="nav-item">
                    <a class="nav-link" href="programType.html">Programs</a>
                  </li>
                  <li class="nav-item">
                    <a class="nav-link" href="classes.html">Classes</a>
                  </li>
                  <li class="nav-item">
                    <a class="nav-link" href="general.parser.html">Parser</a>
                  </li>
                  <li class="nav-item">
                    <a class="nav-link" href="general.fileFormat.html">File Formats</a>
                  </li>
                </ul>
              </div>
                <form class="form-inline my-2 my-lg-0" action="search.html" id="searchTools">
                  <input class="form-control mr-sm-2" placeholder="Search" name="searchTerms" method="GET" value="" type="text" id="searchBox">
                  <button class="btn btn-secondary my-2 my-sm-0" type="submit" id="searchButton">Search</button>
                </form>
              <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#mainNavbar" aria-controls="mainNavbar" aria-expanded="false" aria-label="Toggle navigation" style="">
                <span class="navbar-toggler-icon"></span>
              </button>
            </div>
        </nav>
    </header>
    <main class="container px-0 pt-2 top-buffer">
        <div id="content" class="container">
            <h1 id="cookbook.instrument">Instrument data handling</h1><p></p><p>GROOPS provides functions and programs to read/write, preprocess, analyze and visualize uniformly and non-uniformly sampled instrument data.
This includes tools for filter design and analysis, re-sampling, smoothing, detrending, and power spectrum estimation.
This tutorial goes through exemplary steps for data handling procedures.</p><p><h2>Reading data</h2><p></p><p><ul>

<li>GROOPS is able to read and convert relevant data from various LEO and GNSS satellites. Instrument files need to be converted into the respective GROOPS format using conversion programs.Depending on the content of the input file, the data is stored with a specific <a class="groops-class" href="instrumentTypeType.html">instrument type</a>. User also has the option to change the type later on with <a class="groops-program" href="InstrumentSetType.html">InstrumentSetType</a>.</p><p></li><li>
Multiple files can be concatenated to one file using <a class="groops-program" href="InstrumentConcatenate.html">InstrumentConcatenate</a>. Using this program, it is also possible to sort the epochs, remove the duplicates and NaN values.
</li><li>
<em>Example: Concatenating instrument files</em>
  <ul>

  </li><li>
Create three successively daily sinusoidal signals with <a class="groops-program" href="TimeSeriesCreate.html">TimeSeriesCreate</a> and set their type to MISCVALUE with <a class="groops-program" href="InstrumentSetType.html">InstrumentSetType</a>. In this example, each data set has an overlap of 1 hour with their following dataset.
  </li><li>
Merge all datasets to one single file with <a class="groops-program" href="InstrumentConcatenate.html">InstrumentConcatenate</a>.
  <figure><img class="figure" style="width:80%;" src="../figures/instrumentDataHandeling_concatenating.png" alt="instrumentDataHandeling_concatenating"><figcaption class="center">Figure: Example 1: Concatenating instrument files into one dataset.</figcaption></figure>
  </li></ul>

 <li>Many measurements involve data collected asynchronously by multiple sensors with different sampling. Use <a class="groops-program" href="InstrumentSynchronize.html">InstrumentSynchronize</a> for a continuous harmonization of the data over time or segmentation of the data into arcs.
</li></ul>
</p><p><h2>Preprocessing</h2><p>
Real-world data is often incomplete, inconsistent, and/or lacking in certain behaviors or trends, and is likely to contain many errors. Data preprocessing is a proven method of resolving such issues. Following steps are usually required to be taken:</p><p><ul>

<li>Gross outlier removal:
  <ul>

  </li><li>
Create reference values to compare the input data with. Depending on the instrument type, this can be done by simulation programs such as <a class="groops-program" href="SimulateAccelerometer.html">SimulateAccelerometer</a> or <a class="groops-program" href="SimulateStarCamera.html">SimulateStarCamera</a>. If no reference data is available, the outlier detection is based on the data itself. If needed, synchronize the reference data file and the input data with <a class="groops-program" href="InstrumentSynchronize.html">InstrumentSynchronize</a>.</p><p>  </li><li>
In case of star camera data, compute the differences between the input data and the reference data with <a class="groops-program" href="InstrumentStarCameraMultiply.html">InstrumentStarCameraMultiply</a>.</p><p>  </li><li>
Set a threshold for outlier detection in <a class="groops-program" href="InstrumentRemoveEpochsByCriteria.html">InstrumentRemoveEpochsByCriteria</a>. The threshold is defined empirically according to the accuracy characteristics of each data products. If the differences exceed a predefined threshold, the corresponding epochs are removed. An arbitrary margin can be defined to additionally remove epochs before and after the identified outliers. It is also possible to remove epochs at specific times using <a class="groops-program" href="InstrumentRemoveEpochsByTimes.html">InstrumentRemoveEpochsByTimes</a>.
  </li><li>
Missing epochs can be filled by reference data with <a class="groops-program" href="InstrumentConcatenate.html">InstrumentConcatenate</a>.
  </li><li>
It is also possible to interpolate the missing epochs with <a class="groops-program" href="InstrumentResample.html">InstrumentResample</a>.
  </li><li>
<em>Example: Removing outliers in a synthetic data.</em>
  <ul>

  </li><li>
Create a sinusoidal signal with an amplitude of 1.0 using <a class="groops-program" href="TimeSeriesCreate.html">TimeSeriesCreate</a> and set its type to MISCVALUE with <a class="groops-program" href="InstrumentSetType.html">InstrumentSetType</a>.
  </li><li>
Add zero-mean, white Gaussian noise with a standard deviation of 0.1 with <a class="groops-program" href="NoiseInstrument.html">NoiseInstrument</a>. Interpret this data as a real measurement file.
  </li><li>
Set the threshold criteria to 0.2 in <a class="groops-program" href="InstrumentRemoveEpochsByCriteria.html">InstrumentRemoveEpochsByCriteria</a> and remove the outliers and their nearest epochs in 20 second interval.
  </li><li>
Fill the data gaps with <a class="groops-program" href="InstrumentResample.html">InstrumentResample</a>.
  <figure><img class="figure" style="width:80%;" src="../figures/instrumentDataHandeling_outlierRemoval.png" alt="instrumentDataHandeling_outlierRemoval"><figcaption class="center">Figure: Example 2: Removing gross outliers.</figcaption></figure>
  </li></ul>

  </ul></p><p>
<li>Downsampling:
  <ul>

  </li><li>
If the sampling is irregular use <a class="groops-program" href="InstrumentResample.html">InstrumentResample</a> to make the sampling equidistant.
  </li><li>
Use <a class="groops-program" href="InstrumentSynchronize.html">InstrumentSynchronize</a> to divide the data at gaps into arcs.
  </li><li>
Apply a lowpass filter (e.g. Butterworth) with the Nyquist frequency of the target sampling as cutoff with <a class="groops-program" href="InstrumentFilter.html">InstrumentFilter</a>. Apply the filter in both directions to avoid phase shifts.
  </li><li>
Use <a class="groops-program" href="InstrumentReduceSampling.html">InstrumentReduceSampling</a> to down-sample the data.
  </li></ul>
</p><p><li>Calibration:
  <ul>

  </li><li>
For a general instrument file, <a class="groops-program" href="InstrumentDetrend.html">InstrumentDetrend</a> subtracts offsets or linear/nonlinear trends from the input data. This can be achieved also with <a class="groops-program" href="FunctionsCalculate.html">FunctionsCalculate</a> or <a class="groops-program" href="InstrumentArcCalculate.html">InstrumentArcCalculate</a> by applying determined calibration factors or solving a least-square adjustment.
  </li><li>
For accelerometer data, <a class="groops-program" href="InstrumentAccelerometerEstimateBiasScale.html">InstrumentAccelerometerEstimateBiasScale</a> is designed to estimate and subtract complex biases or scales with respect to simulated accelerometer data. If a thruster file is given, the corresponding epochs are eliminated during estimation process.
  </li><li>
<em>Example: GRACE-C accelerometer calibration</em>
  <ul>

  </li><li>
For one particular date, read and convert Level-1B GRACE-C orbit, star camera, accelerometer, and thruster data with <a class="groops-program" href="GraceL1b2Orbit.html">GraceL1b2Orbit</a>, <a class="groops-program" href="GraceL1b2StarCamera.html">GraceL1b2StarCamera</a>, <a class="groops-program" href="GraceL1b2Accelerometer.html">GraceL1b2Accelerometer</a>, and <a class="groops-program" href="GraceL1b2Thruster.html">GraceL1b2Thruster</a> respectively. It is also required to read the macro-model data of the satellite using the related information in the official document and convert it to GROOPS format with <a class="groops-program" href="SatelliteModelCreate.html">SatelliteModelCreate</a>.
  </li><li>
Use <a class="groops-program" href="SimulateAccelerometer.html">SimulateAccelerometer</a> to generate simulated accelerations due to non-gravitational force models including: <a class="groops-class" href="miscAccelerationsType.html#atmosphericDrag">miscAccelerations:atmosphericDrag</a>, <a class="groops-class" href="miscAccelerationsType.html#solarRadiationPressure">miscAccelerations:solarRadiationPressure</a>, and <a class="groops-class" href="miscAccelerationsType.html#albedo">miscAccelerations:albedo</a>.
  </li><li>
Calibrate the real measurements with a daily constant accelerometer bias by choosing a constant parameter per axis in <a class="groops-class" href="parametrizationAccelerationType.html#accBias">parametrizationAcceleration:accBias</a>.
  <figure><img class="figure" style="width:80%;" src="../figures/instrumentDataHandeling_calibration.png" alt="instrumentDataHandeling_calibration"><figcaption class="center">Figure: Example 3: Calibrating GRACE-C ACT1B data.</figcaption></figure>
  </li></ul>

  </ul>
</ul></p><p><h2>Statistical analysis</h2><p>
<ul>

<li><a class="groops-program" href="InstrumentStatisticsTimeSeries.html">InstrumentStatisticsTimeSeries</a> returns statistics for one or more instrument files. <a class="groops-program" href="InstrumentArcCalculate.html">InstrumentArcCalculate</a> is also able to generate a <strong class="groops-config-element">statistics</strong> file with one mid epoch per arc.
</li></ul>
</p><p><h2>Spectral analysis</h2><p>
Spectral analysis studies the frequency spectrum contained in discrete, uniformly sampled data. The Fourier transform is a tool that reveals frequency components of a signal by representing it in frequency space. The Power Spectral Density (PSD) is a measurement of the energy at each frequency.</p><p><ul>

<li>If the sampling is irregular use <a class="groops-program" href="InstrumentResample.html">InstrumentResample</a> to make the sampling equidistant.
</li><li>
Use <a class="groops-program" href="Instrument2PowerSpectralDensity.html">Instrument2PowerSpectralDensity</a> to compute PSD.
</li><li>
If covariance function of a dataset is available, use <a class="groops-program" href="CovarianceFunction2PowerSpectralDensity.html">CovarianceFunction2PowerSpectralDensity</a>.
</li><li>
<em>Example: Spectral analysis of a synthetic signal.</em>
  <ul>

  </li><li>
Create a sinusoidal signal with an amplitude of 1.0 using <a class="groops-program" href="TimeSeriesCreate.html">TimeSeriesCreate</a> and set its type to MISCVALUE with <a class="groops-program" href="InstrumentSetType.html">InstrumentSetType</a>. Interpret this data as a simulation data file.
  </li><li>
Add zero-mean, white Gaussian noise with a standard deviation of 0.1 with <a class="groops-program" href="NoiseInstrument.html">NoiseInstrument</a>. Interpret this data as a real measurement file.
  </li><li>
Compute PSD of the simulated and measurement data and represent the results with <a class="groops-program" href="PlotGraph.html">PlotGraph</a>.
  <figure><img class="figure" style="width:80%;" src="../figures/instrumentDataHandeling_psd.png" alt="instrumentDataHandeling_psd"><figcaption class="center">Figure: Example 4: Spectral analysis of a synthetic signal.</figcaption></figure>
  </li></ul>

</ul></p><p><h2>Data visualization</h2><p>
<ul>

<li>Argument of latitude plot
Plotting instrument data as a function of satellite position in orbit and time reveals features related to the orbit geometry or environmental conditions. For circular orbits, the position of  satellite can be specified by the argument of latitude.
<ul>

</li><li>
Synchronize the instrument data file with the related orbit data using <a class="groops-program" href="InstrumentSynchronize.html">InstrumentSynchronize</a>.
</li><li>
Use <a class="groops-program" href="Orbit2ArgumentOfLatitude.html">Orbit2ArgumentOfLatitude</a> to compute argument of latitude at each epoch.
</li><li>
Plot the instrument data versus argument of latitude and time with <a class="groops-program" href="PlotGraph.html">PlotGraph</a>.
</li><li>
<em>Example: Argument of latitude representation of GRACE-C eclipse factors</em>
  <ul>

  </li><li>
Compute eclipse factors at each epoch of GRACE-C orbit at an arbitrary time using <a class="groops-program" href="Orbit2EclipseFactor.html">Orbit2EclipseFactor</a>.
  </li><li>
Synchronize the eclipse factor data file with the related orbit data using <a class="groops-program" href="InstrumentSynchronize.html">InstrumentSynchronize</a>.
  </li><li>
Use <a class="groops-program" href="Orbit2ArgumentOfLatitude.html">Orbit2ArgumentOfLatitude</a> to compute argument of latitude at each epoch and visualize the results with <a class="groops-program" href="PlotGraph.html">PlotGraph</a>.
  <figure><img class="figure" style="width:80%;" src="../figures/instrumentDataHandeling_aol.png" alt="instrumentDataHandeling_aol"><figcaption class="center">Figure: Example 5: GRACE-C eclipse factors represented in argument of latitude plot.</figcaption></figure>
  </li></ul>

</ul></p><p><li>Ground-track plot
Plotting instrument data with respect to the satellite ground track is useful to identify any features of geophysical origin in the data.
<ul>

</li><li>
Synchronize the instrument data file with the related orbit data using <a class="groops-program" href="InstrumentSynchronize.html">InstrumentSynchronize</a>.
</li><li>
Use <a class="groops-program" href="Orbit2Groundtracks.html">Orbit2Groundtracks</a> to map instrument data to satellite ground-track.
</li><li>
Visualize the output with <a class="groops-program" href="PlotMap.html">PlotMap</a>.
</li><li>
<em>Example 6: Ground-track representation of GRACE-C eclipse factors</em>
  <ul>

  </li><li>
Compute eclipse factors at each epoch of GRACE-C orbit at an arbitrary time using <a class="groops-program" href="Orbit2EclipseFactor.html">Orbit2EclipseFactor</a>.
  </li><li>
Synchronize the eclipse factor data file with the related orbit data using <a class="groops-program" href="InstrumentSynchronize.html">InstrumentSynchronize</a>.
  </li><li>
Use <a class="groops-program" href="Orbit2Groundtracks.html">Orbit2Groundtracks</a> to generate the gridded data. Each grid value represents the mean value of eclipse factor over the instrument time period (1 month).  visualize the results with <a class="groops-program" href="PlotMap.html">PlotMap</a>.
  </li></ul>

  <figure><img class="figure" style="width:80%;" src="../figures/instrumentDataHandeling_groundTrack.png" alt="instrumentDataHandeling_groundTrack"><figcaption class="center">Figure: Example 6: GRACE-C eclipse factors represented in ground-track plot.</figcaption></figure>
</ul></p><p></ul>

</p>

        </div>
    </main>
</body>
</html>
